CRYPTO-172: Add support for Linux-riscv64 #264
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Java supports RISC-V since version 19. Given Apache Common Crypto is a commonly used library in the Java ecosystem, it's important for it to support RISC-V as well. This patch adds the linux-riscv64 target similarly to linux-aarch64
sebbASF
requested changes
Oct 30, 2023
|
@sebbASF I couldn't find how the test is run on linux-aarch64. Where could I find it? |
|
I don't know, that's why I asked. I assume there is some GH documentation on which architectures are available. |
Run with docker compose -f src/docker/docker-compose.yaml run crypto-riscv64-test
``` $> docker compose -f src/docker/docker-compose.yaml run crypto-riscv64-test WARN[0000] Found orphan containers ([docker-cross-compile-riscv64-build-run-397c7aebdada docker-cross-compile-riscv64-runtime-setup-1 docker-cross-compile-riscv64-build-run-93a982a05cdc docker-cross-compile-riscv64-build-run-7dc6a41f1d5b docker-cross-compile-riscv64-build-run-644e853cbf55 docker-cross-compile-riscv64-build-run-3b09454dc280]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. [+] Running 1/1 ! crypto-riscv64-test Warning 2.0s [+] Building 303.7s (9/9) FINISHED docker-container:festive_mccarthy => [crypto-riscv64-test internal] load build definition from Dockerfile.riscv64 0.0s => => transferring dockerfile: 1.97kB 0.0s => [crypto-riscv64-test internal] load metadata for docker.io/riscv64/ubuntu:20.04 0.7s => [crypto-riscv64-test internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => CACHED [crypto-riscv64-test 1/4] FROM docker.io/riscv64/ubuntu:20.04@sha256:c0e96f609e5f128bdb6d24d21d3bf 0.0s => => resolve docker.io/riscv64/ubuntu:20.04@sha256:c0e96f609e5f128bdb6d24d21d3bf0a9af6f17234976fa574dad7146 0.0s => [crypto-riscv64-test 2/4] RUN apt-get update && apt-get --assume-yes install software-properties-common 274.2s => [crypto-riscv64-test 3/4] RUN curl -L https://dlcdn.apache.org/maven/maven-3/3.8.8/binaries/apache-maven- 1.0s => [crypto-riscv64-test 4/4] WORKDIR /home/crypto 0.0s => [crypto-riscv64-test] exporting to docker image format 27.7s => => exporting layers 17.4s => => exporting manifest sha256:9068927ccfbc46788ec15578541ebc06302b538d4ed75e308b694e1132dd55fd 0.0s => => exporting config sha256:27adda314855e324670f04d2cc656b48c19879ccf16b9e4f32f32cd273bb3319 0.0s => => sending tarball 10.2s => [crypto-riscv64-test crypto-riscv64-test] importing to docker 9.1s Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39) Maven home: /opt/maven Java version: 21-ea, vendor: Private Build, runtime: /usr/lib/jvm/java-21-openjdk-riscv64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "6.2.0-35-generic", arch: "riscv64", family: "unix" [INFO] Scanning for projects... [INFO] [INFO] -----------------< org.apache.commons:commons-crypto >------------------ [INFO] Building Apache Commons Crypto 1.2.1-SNAPSHOT [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-surefire-plugin:3.1.2:test (default-cli) @ commons-crypto --- [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.commons.crypto.CryptoTest OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/commons-crypto-1e590661-aa26-4da1-b5c2-8d721ca08d18-libcommons-crypto.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.095 s -- in org.apache.commons.crypto.CryptoTest [INFO] Running org.apache.commons.crypto.OsInfoTest Linux/riscv64 Linux riscv64 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.663 s -- in org.apache.commons.crypto.OsInfoTest [INFO] Running org.apache.commons.crypto.utils.UtilsTest [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.159 s -- in org.apache.commons.crypto.utils.UtilsTest [INFO] Running org.apache.commons.crypto.utils.EnumTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.033 s -- in org.apache.commons.crypto.utils.EnumTest [INFO] Running org.apache.commons.crypto.jna.CtrCryptoJnaStreamTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.60 s -- in org.apache.commons.crypto.jna.CtrCryptoJnaStreamTest [INFO] Running org.apache.commons.crypto.jna.CtrNoPaddingCipherJnaStreamTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.532 s -- in org.apache.commons.crypto.jna.CtrNoPaddingCipherJnaStreamTest [INFO] Running org.apache.commons.crypto.jna.OpenSslJnaTest Apache Commons Crypto OpenSslJna: enabled = true, version = 0x1010106F OpenSSLVersion(0): OpenSSL 1.1.1f 31 Mar 2020 OpenSSLVersion(1): compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-zF9Nk8/openssl-1.1.1f=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 OpenSSLVersion(2): built on: Tue Oct 10 09:03:48 2023 UTC OpenSSLVersion(3): platform: debian-riscv64 OpenSSLVersion(4): OPENSSLDIR: "/usr/lib/ssl" OpenSSLVersion(5): ENGINESDIR: "/usr/lib/riscv64-linux-gnu/engines-1.1" [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.394 s -- in org.apache.commons.crypto.jna.OpenSslJnaTest [INFO] Running org.apache.commons.crypto.jna.OpenSslJnaCipherTest [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.447 s -- in org.apache.commons.crypto.jna.OpenSslJnaCipherTest [INFO] Running org.apache.commons.crypto.jna.CbcNoPaddingCipherJnaStreamTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.448 s -- in org.apache.commons.crypto.jna.CbcNoPaddingCipherJnaStreamTest [INFO] Running org.apache.commons.crypto.jna.PositionedCryptoInputStreamJnaTest [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.210 s -- in org.apache.commons.crypto.jna.PositionedCryptoInputStreamJnaTest [INFO] Running org.apache.commons.crypto.jna.OpenSslNativeJnaTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.300 s -- in org.apache.commons.crypto.jna.OpenSslNativeJnaTest [INFO] Running org.apache.commons.crypto.jna.OpenSslJnaCryptoRandomTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.24 s -- in org.apache.commons.crypto.jna.OpenSslJnaCryptoRandomTest [INFO] Running org.apache.commons.crypto.jna.CbcPkcs5PaddingCipherJnaStreamTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.609 s -- in org.apache.commons.crypto.jna.CbcPkcs5PaddingCipherJnaStreamTest [INFO] Running org.apache.commons.crypto.cipher.CryptoCipherTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.328 s -- in org.apache.commons.crypto.cipher.CryptoCipherTest [INFO] Running org.apache.commons.crypto.cipher.OpenSslCipherTest [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.684 s -- in org.apache.commons.crypto.cipher.OpenSslCipherTest [INFO] Running org.apache.commons.crypto.cipher.CryptoCipherFactoryTest [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.266 s -- in org.apache.commons.crypto.cipher.CryptoCipherFactoryTest [INFO] Running org.apache.commons.crypto.cipher.OpenSslCommonModeTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s -- in org.apache.commons.crypto.cipher.OpenSslCommonModeTest [INFO] Running org.apache.commons.crypto.cipher.GcmCipherTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.431 s -- in org.apache.commons.crypto.cipher.GcmCipherTest [INFO] Running org.apache.commons.crypto.cipher.JceCipherTest [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.126 s -- in org.apache.commons.crypto.cipher.JceCipherTest [INFO] Running org.apache.commons.crypto.stream.CbcPkcs5PaddingCipherStreamTest [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.579 s -- in org.apache.commons.crypto.stream.CbcPkcs5PaddingCipherStreamTest [INFO] Running org.apache.commons.crypto.stream.output.StreamOutputTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.037 s -- in org.apache.commons.crypto.stream.output.StreamOutputTest [INFO] Running org.apache.commons.crypto.stream.CbcNoPaddingCipherStreamTest [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.528 s -- in org.apache.commons.crypto.stream.CbcNoPaddingCipherStreamTest [INFO] Running org.apache.commons.crypto.stream.input.ChannelInputTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s -- in org.apache.commons.crypto.stream.input.ChannelInputTest [INFO] Running org.apache.commons.crypto.stream.PositionedCryptoInputStreamTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.747 s -- in org.apache.commons.crypto.stream.PositionedCryptoInputStreamTest [INFO] Running org.apache.commons.crypto.stream.CtrNoPaddingCipherStreamTest [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.471 s -- in org.apache.commons.crypto.stream.CtrNoPaddingCipherStreamTest [INFO] Running org.apache.commons.crypto.stream.CtrCryptoStreamTest [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.703 s -- in org.apache.commons.crypto.stream.CtrCryptoStreamTest [INFO] Running org.apache.commons.crypto.random.OpenSslCryptoRandomTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.651 s -- in org.apache.commons.crypto.random.OpenSslCryptoRandomTest [INFO] Running org.apache.commons.crypto.random.CryptoRandomFactoryTest [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.530 s -- in org.apache.commons.crypto.random.CryptoRandomFactoryTest [INFO] Running org.apache.commons.crypto.random.JavaCryptoRandomTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.79 s -- in org.apache.commons.crypto.random.JavaCryptoRandomTest [INFO] Running org.apache.commons.crypto.random.OsCryptoRandomTest [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.594 s -- in org.apache.commons.crypto.random.OsCryptoRandomTest [INFO] Running org.apache.commons.crypto.NativeCodeLoaderTest [WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 1.206 s -- in org.apache.commons.crypto.NativeCodeLoaderTest [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 3 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:18 min [INFO] Finished at: 2023-10-31T13:04:26Z [INFO] ------------------------------------------------------------------------ ```
|
I've added a docker container to run the test and ran it successfully: |
JNA 5.13 on linux-riscv64 introduces dependency on GLIBC 2.34. It's been reported at java-native-access/jna#1557.
|
@sebbASF I've added testing on GitHub Actions for linux-aarch64 and linux-riscv64. You'll note I had to downgrade JNA to 5.12 because of java-native-access/jna#1557. |
|
I would not rollback JNA for ALL configurations just because one has special requirements, just downgrade in that case. |
It's a build-time dependency with the commons-crypto package being the same across all platforms. I'm afraid there is not an easy way to specify per-arch dependency in that case. I'll keep monitoring the JNA linux-riscv64 issue and update anything here whenever there is an update (that we merge this PR with the downgrade first, or that we wait to merge that PR) |
sebbASF
requested changes
Oct 31, 2023
BTW: well done for finding out how to do that! |
Codecov Report
@@ Coverage Diff @@
## master #264 +/- ##
=========================================
Coverage 75.27% 75.27%
Complexity 441 441
=========================================
Files 38 38
Lines 1820 1820
Branches 178 178
=========================================
Hits 1370 1370
Misses 340 340
Partials 110 110 📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today! |
It's been fixed in JNA with java-native-access/jna#1558. Now we just need to wait for a release.
sebbASF
reviewed
Nov 2, 2023
|
The cross-builds now seem to be working, though they do take a long time. Also it looks like they don't only build the native files for their own architectures. |
sebbASF
requested changes
Nov 2, 2023
The step that takes the longest is setting up the java and openssl packages on the aarch64/riscv64 docker images. (ex: https://github.com/apache/commons-crypto/actions/runs/6733654561/job/18304390181#step:4:6004, 6 out of 10 minutes just on that step). This docker image could be cached by pushing it to the repo's docker repository. I've seen that done as part of https://github.com/uraimo/run-on-arch-action. |
sebbASF
reviewed
Nov 2, 2023
Do we need the full JDK and the dev version of libssl? |
It's now taking ~110s vs ~360s previously on my machine
I've trimmed it down to Overall, it goes from ~360s to ~110s locally. And on CI, it goes down to ~150s. |
|
Thanks for your patience dedication getting this together! |
|
@sebbASF Thank you for the review and continuous feedback as well! :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Java supports RISC-V since version 19. Given Apache Common Crypto is a commonly used library in the Java ecosystem, it's important for it to support RISC-V as well.
This patch adds the linux-riscv64 target similarly to linux-aarch64.
I've verified it builds with
docker compose -f src/docker/docker-compose.yaml run crypto src/docker/build.sh.